[WIP] Optimize search and ingestion performance by Copilot · Pull Request #59 · BigSimmo/Database

Copilot · 2026-06-23T04:39:39Z

search-scope.ts: Push metadata filters (sourceStatuses, validationStatuses, extractionQualities, importBatchIds, collections) into SQL using JSONB operators, eliminating the O(N) pagination loop (up to 5 sequential pages of 1000 docs)
worker/main.ts: Refactor uploadAndCaptionImages to classify/caption images in parallel batches instead of sequentially, reducing latency for documents with many images
worker/main.ts: Parallelize embedding calls for chunk embedding fields and additional embedding fields using Promise.all
Run tests to validate changes

Agent-Logs-Url: https://github.com/BigSimmo/Database/sessions/b232a39a-7297-4198-97b0-5db334f255eb

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f1cddc78a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…tests - DocumentViewer: remove || !isConfigured from serverDemoMode init so demo mode only activates via NEXT_PUBLIC_DEMO_MODE=true, not when Supabase is unconfigured (fixes 'document viewer private missing source state is coherent') - ClinicalDashboard: add storedSessionExists check so test-seeded localStorage auth token enables canUsePrivateApis when env is ready (fixes 'duplicate upload warning and exact-copy batch count are visible') Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

# Conflicts: # src/lib/search-scope.ts

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 73a156f63a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

…ers (E-3c PR-C) Dose and monitoring answers must carry the asked-for figure/schedule when a cited chunk verbatim supports it (run-#58 baseline: targeting 1/5 on both intents, every miss = missing figure). Lead-slot promotion swaps/appends the first later figure-bearing fact ONLY when no lead carries one, guarded by the claim-support atom corpus (sourceEvidenceText, now exported) so a promoted figure can never convert a grounded answer into an evidence gap - the guard proven load-bearing by red-testing it disabled. The dose/threshold generation fallback now prefers the safe candidate that carries the intent figure over the first safe one; the safety gate itself is unchanged. Fallback helpers extracted to rag-extractive-answer (cycle-check verified); rag.ts 4908->4901. Six discriminating tests, each verified failing on prior code. Full suite 3068 passed / 1 known container artifact. RAG impact: behaviour change (extractive answer composition + fallback candidate preference only) - E-4 pair vs banked #57/#58/#59 baselines post-merge; retrieval, ranking, selection ordering untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

…ers (E-3c PR-C) (#1039) * docs(ledger): E-3b + PR-B live pair verdict (run #59 green vs #57/#58) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 * feat(rag): figure-aware extractive selection for dose/monitoring answers (E-3c PR-C) Dose and monitoring answers must carry the asked-for figure/schedule when a cited chunk verbatim supports it (run-#58 baseline: targeting 1/5 on both intents, every miss = missing figure). Lead-slot promotion swaps/appends the first later figure-bearing fact ONLY when no lead carries one, guarded by the claim-support atom corpus (sourceEvidenceText, now exported) so a promoted figure can never convert a grounded answer into an evidence gap - the guard proven load-bearing by red-testing it disabled. The dose/threshold generation fallback now prefers the safe candidate that carries the intent figure over the first safe one; the safety gate itself is unchanged. Fallback helpers extracted to rag-extractive-answer (cycle-check verified); rag.ts 4908->4901. Six discriminating tests, each verified failing on prior code. Full suite 3068 passed / 1 known container artifact. RAG impact: behaviour change (extractive answer composition + fallback candidate preference only) - E-4 pair vs banked #57/#58/#59 baselines post-merge; retrieval, ranking, selection ordering untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 * fix(rag): verbatim-corpus check for zero-atom figure promotion (reviewer P2) Both reviewers converged on the gap: bare-number intervals like 'every 6 weeks' match the monitoring figure pattern yet yield no clinical value atom, so the atom-only guard passed them trivially - a figure sourced only from adjacent context could then be promoted and nuked by claim support (safe direction, but converts a shippable answer into an evidence gap). Zero-atom figures now require the matched figure substring verbatim in the citing chunk's claim-support corpus (intentFigureMatchText). Proven both directions by three new tests; comment-precision nit on the corpus lockstep folded in. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 * fix(rag): full-interval figure matching + genuine zero-atom guard coverage CodeRabbit review on the P2 guard: reorder monitoringIntervalFigurePattern so the unit-bearing "every N unit" alternative precedes bare "every N" — exec() now returns the full schedule instead of a truncated "every 6". Digit+unit intervals turn out to produce quantity atoms (6/week), so atom identity already refuses unit mismatches; the reorder hardens the verbatim fallback against atom-coverage drift, and both comments claiming "every 6 weeks" is zero-atom are corrected. The non-discriminating first guard test is replaced with a mismatched-unit pin (shared number, different unit, figure only in adjacent context) plus two genuinely zero-atom tests ("annually") that exercise the verbatim-corpus path both directions. promotionAtomKey hand-mirror replaced by exporting clinicalValueAtomKey from rag-claim-support. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --------- Co-authored-by: Claude <noreply@anthropic.com>

…59) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

Initial plan

d53a288

Copilot AI assigned Copilot and BigSimmo Jun 23, 2026

Copilot started work on behalf of BigSimmo June 23, 2026 04:59 View session

Copilot AI added 2 commits June 23, 2026 05:04

chore: planning optimization changes

6d4454c

Changes before error encountered

ae039b5

Agent-Logs-Url: https://github.com/BigSimmo/Database/sessions/b232a39a-7297-4198-97b0-5db334f255eb

Copilot stopped work on behalf of BigSimmo due to an error June 23, 2026 05:06
402 {"error":{"message":"You have exceeded your monthly quota","code":"quota_exceeded"}} (Request ID: 2C00:136354:14D1F97:16F41BF:6A3A145C)

Copilot AI requested a review from BigSimmo June 23, 2026 05:06

BigSimmo added 2 commits June 23, 2026 13:51

Merge branch 'main' into copilot/optimize-search-ingestion-performance

46a2206

Merge branch 'main' into copilot/optimize-search-ingestion-performance

f1cddc7

BigSimmo marked this pull request as ready for review June 23, 2026 05:54

chatgpt-codex-connector Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread src/lib/search-scope.ts Outdated

Comment thread src/lib/search-scope.ts Outdated

Comment thread src/lib/search-scope.ts Outdated

Add paginated scope resolution and robust fallback filters

eaf58ab

Copilot AI review requested due to automatic review settings June 26, 2026 05:52

Copilot AI reviewed Jun 26, 2026

BigSimmo and others added 2 commits June 26, 2026 16:19

Merge remote-tracking branch 'origin/main' into pr59-fix

73a156f

# Conflicts: # src/lib/search-scope.ts

chatgpt-codex-connector Bot reviewed Jun 26, 2026

View reviewed changes

Comment thread src/components/ClinicalDashboard.tsx

Comment thread src/lib/search-scope.ts

Comment thread src/lib/search-scope.ts

Comment thread src/components/ClinicalDashboard.tsx

BigSimmo enabled auto-merge June 26, 2026 12:21

BigSimmo merged commit d6878e5 into main Jun 26, 2026
4 checks passed

BigSimmo deleted the copilot/optimize-search-ingestion-performance branch June 27, 2026 05:14

BigSimmo pushed a commit that referenced this pull request Jul 21, 2026

docs(ledger): E-3b + PR-B live pair verdict (run #59 green vs #57/#58)

d50b465

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

BigSimmo mentioned this pull request Jul 21, 2026

feat(rag): figure-aware extractive selection for dose/monitoring answers (E-3c PR-C) #1039

Merged

12 tasks

BigSimmo pushed a commit that referenced this pull request Jul 21, 2026

docs(ledger): E-4 pair verdict — E-3 wave adopted (run #60 vs #57/#58/#…

d834dec

…59) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Optimize search and ingestion performance#59

[WIP] Optimize search and ingestion performance#59
BigSimmo merged 8 commits into
mainfrom
copilot/optimize-search-ingestion-performance

Copilot AI commented Jun 23, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jun 23, 2026 •

edited

Loading